Motion Assessment Using a Game Controller

ABSTRACT

A technique is disclosed for motion assessment of an individual using a game controller in connection with interactive media, e.g. a multiplayer game, such as Simon says, where game play is based, at least in part, upon coordinated or directed physical interaction of multiple players.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationSer. No. 61/043,696, filed 9 Apr. 2008, which application isincorporated herein in its entirety by this reference thereto.

FIELD OF THE INVENTION

Embodiments of this invention relate to interactive media. Moreparticularly, embodiments of this invention relate to motion assessmentof an individual using a game controller in connection with interactivemedia.

BACKGROUND OF THE INVENTION

Simon says is a game for three or more players, most often children. Oneof the people is “it”—i.e. Simon. The others must do what Simon tellsthem to do when asked with a phrase beginning with “Simon says.” IfSimon says “Simon says jump,” then the other players must jump. Thoseplayers who do not jump are out. However, if Simon says simply “jump,”without first saying “Simon says,” players do not jump. Those who dojump are out. In general, it is the spirit of the command, not theactions that matters. If Simon says “Simon says touch your toes,”players only have to show that they are trying to touch their toes. Itis the ability to distinguish between valid and invalid commands, ratherthan physical ability, that matters.

It is Simon's task to try to get everyone out as quickly as possible,and it is every one else's job to stay in for as long as possible. Thelast of Simon's followers to stay in wins.

Simon says is an example of a game that is played by a group of people,where each player is present at the same place at the same time. Gamesof this type are therefore not amenable to distributed play, for examplewith players at various remote locations, because there is no known wayof coordinating and assessing each player's movement, for example, inresponse to a command from Simon in the Simon says game.

Thus, there is a need in the art, for a technique that assesses usermotion in multiplayer games, where such games require physicalinteraction of the users although the players are at differentlocations.

SUMMARY OF THE INVENTION

The disadvantages associated with the prior art are overcome byembodiments of the present invention directed to a technique for motionassessment of an individual using a game controller in connection withinteractive media. In an embodiment, the media may be a multiplayergame, such as Simon says, where game play is based, at least in part,upon coordinated or directed physical interaction of multiple players.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the present invention can be readily understood byconsidering the following detailed description in conjunction with theaccompanying drawings, in which:

FIG. 1 is a block schematic diagram of a system for motion assessmentpursuant to game play according to an embodiment of the invention; and

FIG. 2 is a block schematic diagram of a game server according to anembodiment of the invention.

DESCRIPTION OF THE SPECIFIC EMBODIMENTS

Although the following detailed description contains many specificdetails for the purposes of illustration, anyone of ordinary skill inthe art will appreciate that many variations and alterations to thefollowing details are within the scope of the invention. Accordingly,the exemplary embodiments of the invention described below are set forthwithout any loss of generality to, and without imposing limitationsupon, the claimed invention.

An embodiment of the invention is directed to a technique for motionassessment of an individual using a game controller in connection withinteractive media. In an embodiment, the media may be a multiplayergame, such as Simon says, where game play is based, at least in part,upon coordinated or directed physical interaction of multiple players.

FIG. 1 is a block schematic diagram of a system for motion assessmentpursuant to game play according to an embodiment of the invention. InFIG. 1, a plurality of participant locations are linked by a network 14,such as the Internet. In the example of FIG. 1, the game of Simon saysinvolves a master site 10, where one or more individuals 13 issueinstructions, which are captured via a motion assessment device 11, suchas a video camera and game console. Those skilled in the art willappreciate that other motion assessment mechanisms are available andcould be used in connection with the invention, for example, ultrasonicmotion detectors, infrared motion detectors, laser motion detectors,hand-held motion detectors, motion detectors that are worn on the body,and combinations thereof. Generally, the participant who isdemonstrating a motion is best assessed with a video camera 19. Thisallows demonstrated motions to be exhibited on displays 18 a-18 n atvarious participant sites. A display 18 may also be provided at themaster site to allow the master to observe actions of the participantsand/or to observe demonstrated actions, should the master be moved toanother site. However, commands may be provided verbally, rather thanbeing demonstrated, or the demonstrated movements may be stored locallyat each of the participant sites and may be displayed upon receipt of aselection command from a master participant.

A plurality of participant sites 12 a-12 n allow one or moreparticipants 17 a-17 n to respond to such instruction pursuant to gameplay. Actions of the participants at these sites are assessed via amotion assessment device 15 a-15 n, as discussed above which, in thisembodiment, includes a video camera 19 a-19 n. In this embodiment,participant site12 n assesses motion with devices 34 that are worn oneach participant's wrist.

In this embodiment, a game server 16 coordinates game play, scoring, andcan also provide image and voice recognition functions and othercapability. Other embodiments rely upon a game console for suchfunctions, while still other embodiments do not rely upon a gameconsole, but provide all image and display information directly to agame server. Further, it is not necessary for each participant to usethe same motion assessment mechanism. Thus, some locations may employhand-held devices, while others may employ video cameras, etc.

An embodiment of the invention uses a video camera, such as the SonyPlayStation Eye, which is a digital camera device that operates inconjunction with the PlayStation 3 video game console. Interaction ofgame players may also be captured with a game controller or otherdevice. The video game console and/or a game server operates to recordmovements between players or any entity, as detected by the videocamera, game controller, or other modality, and to compare thosemovements between people at remote locations, for example on other sidesof the Internet, as well a detecting multiple individuals at a samelocation and tracking their motions separately pursuant to game play.Other embodiments of the invention contemplate any other video cameraand game console or game console equivalent, such as the Sony EyeToy andPlayStation 2. Additionally, and as discussed above, other capturemodalities may be used in conjunction with the video camera, such as amotion detector, which may be an ultrasonic motion detection device, amagnetic motion capture device, an RF or IR motion capture device, apressure sensitive device, such as an active floor mat, a device thattracks gravitational forces and/or acceleration, or a device that isheld in the hand or attached to the game player's body, such as the SonySIXAXIS wireless controller, or the like. Further, a device specificallydesigned for capturing motion images in three dimensions may be used,such as an array multiple, spaced cameras, a laser guided device, or thelike.

FIG. 2 is a block schematic diagram of a game server according to anembodiment of the invention. In FIG. 2, a network interface 30 providescommunications to and from the game server. This embodiment is intendedto operate in connection with a system of the type shown in FIG. 1.Those skilled in the art will appreciate that the elements in the gameserver discussed below are not all necessary to practice the invention.The network interface receives voice information, such as voicecommands, and video information, such as captured images of participantsin the game, from each game location.

The voice information is routed to a voice processor 29 that identifiesspoken commands, such as “Hands on head,” and looks up a pose for suchcommand in a pose database 20. The voice information may also be routedto the various participant locations via a routing mechanism 31.

The video information is routed to a video processor 28, which performsthe functions of accessing the pose database pursuant to operation of apose capture module 21. The video processor also provides videoinformation to the routing mechanism for viewing at each participantlocation. An image extraction module 27 operates under supervision ofthe video processor and in conjunction with a player database 22 toidentify participants and, in conjunction with the pose capture moduleand pose database, to identify participant actions. Pose informationfrom the pose database is provided to a compare module 26 under controlof the video and/or voice processors, where a pose template is comparedwith an extracted image and a determination of accuracy, etc. is madewith reference to standards stored in a metrics database 23. A result ofthe comparison is provided to a scoring module 25. Scoring results areboth stored and retrieved from a score database 24 for display to theparticipants with regard to game play. The score results and imageinformation are combined at a composite module 32 and thence routed tothe game participants.

In an embodiment that implements the classic game of Simon says, kids(and/or others) play Simon says against one another. At least one gameplayer, who may be a master, i.e. “it,” and who is chosen randomly orwho is chosen by some other scheme, describes a motion to be undertakenby the other game players. The master's commands are routed to allparticipants, for example via a network connection. The master may thenmake some motion in front of the video camera, which is captured andanalyzed by, for example the game console and/or game server. When otherkids try to do that movement, those movements are also captured, forexample, by a video camera, and a component of the game console and/orgame server programming compares the captured movements against those ofthe master to determine whether or not the master's movement wasmimicked correctly or close enough to qualify as a successful action.The action may also be judged on the timeliness of the participantresponse, e.g. who was first, how fast was the response, did a time-outoccur before the action was completed? The system may also simplyobserve and determine whether the game participants executed the correctmotion, based upon, for example, verbal commands from the master.

One method of tracking the participant comprises processing imagescaptured by, for example, the video camera and identifying both theidentity of the participant, if there are multiple participants at alocation, by recognizing, for example, each participant's face; and bytracking movement each participant's head, arms, legs, and torso, toidentify, for example, a pose of the participant's body for purposes ofcomparison with that of a master and/or an arc of movement of theparticipant's body, for example, to identify compliance of theparticipant with more complex movements.

In an embodiment, a microphone associated with the video camera orotherwise situated in connection with at least a master, may be providedto capture user commands and, for example, via voice recognition,translate such commands into signals that are used to look up apre-recorded motion template. For example, if the master says “Simonsays put your hands on your head,” then a voice recognition moduletranslates this command into a signal that looks up a “hands on head”template in a pose database. The pose captured from each of theparticipants is compared to the template of the pose and compliance isthen determined and recorded for each participant. In this embodiment,the master need not demonstrate the pose, but merely commands that apose be assumed. While these commands are captured for purposes oftemplate look-up, as described above, a mechanism, such as a network, isalso provided for routing these commands to each participant to allowthem to be heard at each location.

Further, while the video camera is used to capture images of theparticipant's for analysis by the game console and/or game server, theimages may also be provided to the master, for example to providecomments upon observed poses and the like. Also, if a remote classroomsetting is provided, the game console and/or game server may beprogrammed to determine if a participant is not present or has left theview of the camera. In this way, an embodiment of the invention may beused to take class attendance and determine if each participant isindeed participating. This aspect of the invention may be used, not onlyin a game setting, but also in an instructional or educational setting,as discussed below.

The game console and/or game server may include programming thatperforms such motion-capture video analysis. While face recognition orother schemes for identifying individuals within a group may be appliedin those embodiments where multiple individuals participate at a singlelocation, it will be appreciated by those skilled in the art that eachindividual in the group may wear an identification device, such as anarticle of clothing that is a particular color and/or that has aparticular marking; or an electronic bracelet or other such signalingdevice may be worn by each individual. In those cases where a singleindividual is present at a location, it is only necessary to know thelocation, for example by a network address, to identify the individualfor purposes of tracking the individual's movements and scoring theindividual's execution of these movements. Other embodiments contemplatethe use of voice recognition to identify participants.

Likewise, a group's reaction can be captured, analyzed and scored. Thus,if Simon says “Put your hands on your head,” the system could determinehow many participants within a group reacted correctly. The group couldbe scored on this basis. In this way, groups of players could competeagainst other groups of players, e.g. the 4^(th) grade class couldcompete against the 5^(th) grade class, the boys could compete againstthe girls, the students could compete against the faculty, etc.

Another embodiment provides educational and/or tutorial activities, forexample where a pre-recorded demonstration of a desired movement by ateacher or other instruction is presented to one or more sessionparticipants, rather than having a master participant in real-time, asin the Simon says example above. Such instruction may be, for example,how to execute a golf swing, a tennis stroke, or other complex motion. Aparticipant in such session views a demonstration of the motion and/orother instructional information, for example, on a television or otherdisplay. A video camera or other device captures the participant'smotions, analyzes and compares the captured motions with an idealmotion, and is then judged or graded on how well he mimicked the actionthat was demonstrated to him.

An embodiment of the invention provides a reference gesture that, forexample, is generated by a live person, e.g. over the Internet, whoperforms an action. The action that the user is performing may bedisplayed to the other participants by a video clip of the person doingit, i.e. the system records the person demonstrating the gesture andthen it shows it as a video clip. The camera can also capture suchgestures from multiple angles, or multiple cameras can be used, suchthat a plurality of views from the plane of the camera can be combinedto provide a 3D view that allows each participant to see allperspectives of what the action is, whether it is for a Simon says game,a golf swing or tennis serve analyzer, or for learning dance moves.

An embodiment of the invention provides each participant with theability to speak and move an object. Thus, each participant can create alogin based on, for example, a unique manipulation of a game controller.

An embodiment comprehends gesture recording, where the gesture occurs infree space. A scoring aspect of invention determines what tolerance aparticipant's gesture achieves relative to a reference gesture, e.g. asnapshot. The game console and/or game server include programming thatis used to compare differences between a reference gesture which may becaptured, e.g. from a 3D tracking device, from a camera, or acombination of both, or which may be pre-recorded, e.g. an expertgesture, and some change over time for some purpose. Depending on thechange, results are provided, such as the person moved the wrong way; orin a golf swing, from the time they start the swing to the time they endthe swing, there is some change over time. For example, with regard to agolf swing, the gesture, i.e. the swing would be decomposed into variouscritical elements, where certain parts of the motion are more importantthan other parts of the motion. Thus, an embodiment of the inventionprovides a weighting of the critical elements within a movement orgesture, such as a golf swing, and relative to a reference, a person'sperformance is scored based on the weighting of various elements withinan actual captured image.

An embodiment provides a library of functions, such as various types ofgolf swing or tennis serves. Such motions are all pre-computed and thevideo camera is used to capture a person's execution relative to thereference in the library to identify how much the person's performancedeviates from the idealized motion. Further, an expert could watch theperson execute a movement in real time and provide instruction andperformance evaluation based upon analysis of a captured movementrelative to a reference movement. In this example, the expert may be aninstructor at a remote location who is observing the person'sperformance with the video camera at the person's location via anetwork.

Likewise, several individuals can demonstrate a skill simultaneously andbe judged and ranked accordingly. For example, there could be an on-linegolf swing competition in which each individual is observed and gradedby the system. Results and ranking would then be exhibited on eachparticipant's display. A human expert may also participant in thejudging activity.

An embodiment starts game play when a participant makes a particulargesture that is recognized by the system as a sign that game play is tocommence. Such gesture can be specified in advance by the gamedeveloper, or it can be recorded by a participant, for example in theSimon says game, a participant, perhaps the master, would lift his handsabove his head to start a game play session. This aspect of theinvention thus concerns performing an action and determining whether ornot the action was taken, e.g. did the participants commence game play?

One embodiment comprises a game in which an initial position isestablished for a player by another player, where the player who isbeing instructed must follow the instructions of the other player. Theperson following the instructions has a reference picture taken on theinitial position, and then after a command is issued by the instructingplayer, a further image is taken or further information is gathered andcompared against the reference picture in real time to determine if itwas performed according to some critical performance characteristic,e.g. quickly enough, accurately enough, or should not have beenperformed at all. As with Simon says, one person may be the leader andother participants follow some form of instruction, imperative command,or illustration of an action, and the participants then respondaccordingly.

An embodiment of the invention may also be used in games that employobjects, as well as individual participants. For example, Hullabaloo(Cranium, Inc, Seattle, Wash.) is a talking game that is played withdifferent colored cards. The cards are placed on the floor and the gameinstructs the game players where to move, often entangling the players,one with the other. An embodiment of the invention, in this vein, usesphysical objects at each participant's location that are representedicons on a display as well. The objects are placed on the floor and theparticipants physically move to the objects as instructed by the icons.The invention tracks the participants, as discussed above, over anetwork based system.

In addition to tracking the individual, game elements could be placed ina room and the system uses the video camera or other such expedient toknow the position of the game elements as well. Thus, an embodimentprovides for, and tracks, both the players who are in the room, wherethe system knows their position, and the game elements that are a partof the play. Another embodiment identifies objects that are in a room,such as a table or chair, and assigns them a role in a game. Forexample, a participant may be instructed to sit on the chair or to liedown on the rug.

While the above is a complete description of the preferred embodiment ofthe present invention, it is possible to use various alternatives,modifications and equivalents. Therefore, the scope of the presentinvention should be determined not with reference to the abovedescription but should, instead, be determined with reference to theappended claims, along with their full scope of equivalents. Any featuredescribed herein, whether preferred or not, may be combined with anyother feature described herein, whether preferred or not. In the claimsthat follow, the indefinite article “A”, or “An” refers to a quantity ofone or more of the item following the article, except where expresslystated otherwise. The appended claims are not to be interpreted asincluding means-plus-function limitations, unless such a limitation isexplicitly recited in a given claim using the phrase “means for.”

1. An apparatus for motion assessment of an individual in connectionwith interactive media, comprising: a plurality of participantlocations; a video camera at each participant location for capturingmovements of each participant at each said participant location; and aprocessor programmed to execute instructions for comparing said capturedmovements of each participant at each of said participant locationsagainst any of a reference movement, an expected or elicited movement,or a movement of one or more other participants.
 2. The apparatus ofclaim 1, said interactive media comprising: a multiplayer game in whichgame play is based, at least in part, upon coordinated or directedphysical interaction of multiple players.
 3. The apparatus of claim 1,said interactive media comprising: a game in which at least one saidparticipant location comprises a master site from which one or moreparticipants issue instructions and/or demonstrates movements, which arecaptured via a video camera at said master site and communicated to eachother of said participant locations; said each other of said participantlocations comprising game play sites at each of which one or moreparticipants respond to said instructions pursuant to game play, andwherein actions of participants at said participant locations arecaptured via a video camera at a corresponding game play site andcommunicated to said processor and/or said master site.
 4. The apparatusof claim 2, further comprising: a game server, in communication withsaid plurality of participant locations via a network, programmed toexecute instructions for any of coordinating game play, scoring gameplay, and providing image and voice recognition functions.
 5. Theapparatus of claim 1, further comprising: at least one additional motioncapture modality comprising any of an ultrasonic motion detectiondevice, a magnetic motion capture device, an RF or IR motion capturedevice, a pressure sensitive device, a device that tracks gravitationalforces and/or acceleration, a device that is held in the hand orattached to a participant's body, and a device for capturing motionimages in three dimensions.
 6. The apparatus of claim 1, said processorprogrammed to execute instructions for detecting multiple participantsat a same participant location and tracking their motions separatelypursuant to game play.
 7. The apparatus of claim 1, said processorprogrammed to execute instructions for receiving and executing voicecommands received from one or more participants via a microphone a saidor more participant's locations.
 8. The apparatus of claim 1, saidprocessor programmed to execute instructions for any of evaluating,critiquing, ranking, and scoring said captured movements, for any ofeach individual participant, groups of participants, and pursuant to acompetition between two or more participants.
 9. The apparatus of claim1, said processor programmed to execute instructions for identifyingboth the identity of a participant, if there are multiple participantsat a location, by recognizing each participant's face; and by trackingmovement each participant's head, arms, legs, and torso, to identify apose of the participant's body for purposes of comparison with that of amaster and/or an arc of movement of the participant's body to identifycompliance of the participant with more complex movements.
 10. Theapparatus of claim 1, said processor programmed to execute instructionsfor determining if a participant is not present at a participantlocation or has left the view of a camera at said participant location.11. The apparatus of claim 1, said processor programmed to executeinstructions for performing any of motion-capture video analysis. 12.The apparatus of claim 1, said processor programmed to executeinstructions for demonstrating a desired movement or other instructionto one or more participants, for analyzing and comparing capturedmotions of said one or more participants responsive to saiddemonstrating relative to an ideal motion, and for judging and/orgrading how well said one or more participants mimicked the movementthat was so demonstrated.
 13. The apparatus of claim 12, wherein saidmovement is a complex movement; and wherein portions of said movementare weighted relative to their difficulty, and/or accuracy and/orprecision required.
 14. The apparatus of claim 1, said processorprogrammed to execute instructions for accessing a library of motiontemplates.
 15. The apparatus of claim 1, said processor programmed toexecute instructions for using physical objects at each participantlocation pursuant to interaction with said interactive media and/or foridentifying and/or tracking physical objects are each participantlocation.
 16. A computer implemented method for motion assessment of anindividual in connection with interactive media, comprising the stepsof: providing a plurality of participant locations; capturing movementsof each participant at each said participant location with a videocamera; and comparing said captured movements of each participant ateach of said participant locations against any of a reference movement,an expected or elicited movement, or a movement of one or more otherparticipants.
 17. The method of claim 16, said interactive mediacomprising: a multiplayer game in which game play is based, at least inpart, upon coordinated or directed physical interaction of multipleplayers.
 18. The method of claim 16, said interactive media comprising:a game in which at least one said participant location comprises amaster site from which one or more participants issue instructionsand/or demonstrates movements, which are captured via a video camera atsaid master site and communicated to each other of said participantlocations; said each other of said participant locations comprising gameplay sites at each of which one or more participants respond to saidinstructions pursuant to game play, and wherein actions of participantsat said participant locations are captured via a video camera at acorresponding game play site and communicated to said processor and/orsaid master site.
 19. An electronic storage medium containing computerinstructions for executing a method for motion assessment of anindividual in connection with interactive media, comprising the stepsof: providing a plurality of participant locations; capturing movementsof each participant at each said participant location with a videocamera; and comparing said captured movements of each participant ateach of said participant locations against any of a reference movement,an expected or elicited movement, or a movement of one or more otherparticipants.